Throubleshoot the startup sequence

Introduction

The most tricky or the most critical bugs are often hidden in the application startup sequence or application static methods.
As an example, startup configurations such as connection string or resources access keys may be wrong or missing.
Also, static contructors within the application or its dependencies may hide tricky bugs that are difficult to troubleshoot.

Those places are usually difficult to troubleshoot as telemetry may be not active when they are executed.

Diginsight telemetry enables full observability also for these parts by means of the DeferredLoggerFactory that provides recording the application flow until the telemetry infrastructure is set up.

Upon setup completion, telemetry recording is flushed right before the standard telemetry flow gathering so that any configuration problem or error can be made visible.

The code snippets below are available as working samples within the telemetry_samples repository.

In particular the steps shown below explain how the startup sequence can be made observable on the SampleWebApi within the telemetry_samples repository.

alt text

How to make the startup sequence observable

The following code snippet shows the startup method of the SampleWebApi project:

public static IDeferredLoggerFactory DeferredLoggerFactory;
internal static readonly ActivitySource ActivitySource = new(typeof(Program).Namespace ?? typeof(Program).Name!);

public static void Main(string[] args)
{
    DiginsightActivitiesOptions activitiesOptions = new() { LogActivities = true };
    DeferredLoggerFactory = new DeferredLoggerFactory(activitiesOptions: activitiesOptions);
    var logger = DeferredLoggerFactory.CreateLogger<Program>();

    ActivitySource activitySource = new(typeof(Program).Namespace!);
    DeferredLoggerFactory.ActivitySources.Add(activitySource);
    DiginsightDefaults.ActivitySource = activitySource;

    IWebHost host;
    using (var activity = DiginsightDefaults.ActivitySource.StartMethodActivity(logger, new { args }))
    {
        host = WebHost.CreateDefaultBuilder(args)
            .ConfigureAppConfiguration2()
            .UseStartup<Startup>()
            .ConfigureServices(services =>
            {
                var logger = DeferredLoggerFactory.CreateLogger<Startup>();
                using var innerActivity = ActivitySource.StartRichActivity(logger, "ConfigureServicesCallback", new { services });

                services.TryAddSingleton(DeferredLoggerFactory);
            })
            .UseDiginsightServiceProvider()
            .Build();

        logger.LogDebug("Host built");
    }

    host.Run();
}

a IDeferredLoggerFactory and an ILogger instance are created at startup, immediatly after application start:

DiginsightActivitiesOptions activitiesOptions = new() { LogActivities = true };
DeferredLoggerFactory = new DeferredLoggerFactory(activitiesOptions: activitiesOptions);
var logger = DeferredLoggerFactory.CreateLogger<Program>();

Dotnet logging is not configured or activated already, however logger instance is recording all activities START, END and all explicit logging operations.

Please, note that the logger factory is passed the activitySources that are used so that it can register as a listener to them.

DeferredLoggerFactory.ActivitySources.Add(activitySource);

the Startup ConfigureServices method registers observability with AddObservability() method.
Also, it registers recorded telemetry flush with FlushOnCreateServiceProvider().

public void ConfigureServices(IServiceCollection services)
{
    services.AddHttpContextAccessor();
    services.AddObservability(configuration);
    services.AddDynamicLogLevel<DefaultDynamicLogLevelInjector>();
    services.FlushOnCreateServiceProvider(deferredLoggerFactory);

After services configuration, right before the host build, the telemetry and logging will be configured and activated. The UseDiginsightServiceProvider() call will run the registered telemetry Flush() and all the deferred logs will be flushed to the telemetry targets.

alt text

Api startup will then show all logs recorded during startup:

alt text
Back to top